Assessment of medical students' Surgery knowledge based on Progress Test.

Progress Testing (PT) is an assessment tool whose use has grown throughout Brazil in the last decade. PT makes it possible to assess the students' knowledge gain throughout the undergraduate course and, for their interpretations to be valid, their items (questions) must have adequate quality from the point of view of content validity and reliability of results. In this study, we analyzed the psychometric characteristics of the items and the performance of students in the content area of surgery from 2017 to 2023. For the analyses, we used the assumptions of Classical Test Theory, Bloom's taxonomy and Cronbach's alpha reliability coefficient. The items were easy (average difficulty index between 0.3-0.4), with fair to good discrimination (discrimination index between 0.3-0.4) and with a predominance of medium to high taxonomy. Reliability remained substantial over the years (>0.6). Students' knowledge gain in surgery was found to be progressive and more important from the 3rd year of the undergraduate course, reaching approximately 70-75% in the 6th year. This measurements framework can be replicated in other contexts for a better understanding of student learning and for qualification of evaluation processes.

Can trauma leagues contribute to better cognitive performance and technical skills of medical students? The experience of the Unicamp trauma league.

PURPOSE: Trauma leagues (TLs) are extracurricular programs that offer medical students supervised exposure to trauma and acute care surgery, mentorship, and participation in other academic activities. TLs are fully approved medical schools, and currently, over 100 TLs exist in Brazil. We hypothesized that the performance/competence of medical students who participated in TLs was superior compared to non-participants. This study evaluated and compared the cognitive performance and technical skills of the two groups. METHODS: This retrospective cohort study evaluated the performance of TL medical students to non-TL alumni from 2005 to 2017, using the students' academic performance coefficient, Clinical Competence Assessment, and Progress Test results. SigmaPlot 12.0 software was used to perform statistical analyses, including Mann-Whitney comparison tests and the Kruskal-Wallis test to confirm the data. RESULTS: Of the 1366 medical students who graduated from a Brazilian university, 966 were included, with 17.9% having participated in TL. Compared to non-TL participants, TL students demonstrated better cognitive performance according to the performance coefficient (p = 0.017) and Progress Test result (p < 0.001), and higher achievement in the Clinical Competence Assessment (p < 0.001). CONCLUSION: The academic performance of TL students was superior to that of non-TL students at the University of Campinas (Unicamp), suggesting a positive impact of TL in the preparation of future doctors. The study findings suggest that participation in TL at Unicamp was beneficial in preparing better doctors and should be considered by medical schools worldwide. EVIDENCE LEVEL: II (Retrospective cohort).

Assessment of medical students' Surgery knowledge based on Progress Test / Avaliação do conhecimento de estudantes de medicina na área de Cirurgia a partir do Teste de Progresso

ABSTRACT Progress Testing (PT) is an assessment tool whose use has grown throughout Brazil in the last decade. PT makes it possible to assess the students' knowledge gain throughout the undergraduate course and, for their interpretations to be valid, their items (questions) must have adequate quality from the point of view of content validity and reliability of results. In this study, we analyzed the psychometric characteristics of the items and the performance of students in the content area of surgery from 2017 to 2023. For the analyses, we used the assumptions of Classical Test Theory, Bloom's taxonomy and Cronbach's alpha reliability coefficient. The items were easy (average difficulty index between 0.3-0.4), with fair to good discrimination (discrimination index between 0.3-0.4) and with a predominance of medium to high taxonomy. Reliability remained substantial over the years (>0.6). Students' knowledge gain in surgery was found to be progressive and more important from the 3rd year of the undergraduate course, reaching approximately 70-75% in the 6th year. This measurements framework can be replicated in other contexts for a better understanding of student learning and for qualification of evaluation processes.

RESUMO O Teste de Progresso (TP) é uma ferramenta de avaliação cujo uso tem crescido em todo o Brasil na última década. O TP permite avaliar o ganho de conhecimento dos estudantes ao longo do curso de graduação e, para que suas interpretações sejam válidas, é preciso que seus itens (questões) tenham qualidade adequada do ponto de vista de validade de conteúdo e confiabilidade de resultados. Neste estudo, analisamos as características psicométricas dos itens e o desempenho dos estudantes na área de cirurgia do TP de 2017 a 2023. Para as análises, usamos os pressupostos da Teoria Clássica dos Testes, a taxonomia de Bloom e o coeficiente de fidedignidade alfa de Cronbach. Os itens se mostraram fáceis (índice de dificuldade média entre 0,3-0,4), com discriminação de regular a boa (índice de discriminação entre 0,3-0,4) e com predomínio de questões de média a alta taxonomia. A confiabilidade se manteve substancial ao longo dos anos (>0,6). O ganho de conhecimento dos estudantes em cirurgia é progressivo e mais importante a partir do 3º ano do curso de graduação, chegando a aproximadamente 70-75% no 6º ano. Este arcabouço de aferições pode ser replicado em outros contextos para melhor compreensão do aprendizado dos estudantes e para qualificação dos processos avaliativos.

Ensino interprofissional em saúde: análise qualitativa da experiência de estudantes, preceptores e tutores / Interprofessional teaching in health: qualitative analysis of the experience of students, preceptors and tutors / Enseñanza interprofesional en sanidad: análisis cualitativo de la experiencia de estudiantes, preceptores y tutores

A reorientação da formação profissional por meio de políticas públicas ofereceu a oportunidade para a criação de componentes curriculares, para formar estudantes que exercitem práticas interprofissionais. Este trabalho faz uma análise do contexto e implicações sobre a implantação de uma disciplina interprofissional nos cursos da saúde em uma universidade pública. Foi realizado um estudo com perfil exploratório de abordagem qualitativa utilizando-se grupos focais constituídos por tutores, preceptores e estudantes. Definidas as categorias e seus descritores foi utilizado o modelo 3P para a análise de conteúdo e dos resultados. Vale destacar a ressignificação do papel docente, a percepção dos estudantes frente aos desafios do ensino interprofissional e a aproximação com o serviço de saúde. Correlacionamos na discussão geral deste texto linhas de discussão junto à assistência à pessoa, à sociologia e à pedagogia. A inovação proposta pela disciplina interprofissional de Atenção em Saúde seguiu o ideário de uma agenda internacional. Os relatos de tutores, preceptores e estudantes foram um estímulo decisivo para prosseguir avançando nas demais séries dos cursos, em atividades colaborativas e práticas interprofissionais.

The reorientation of professional training through public policies offered the opportunity for the creation of curricular components to train students to exercise interprofessional practices. This paper makes an analysis of the context and implications on the implementation of an interprofessional discipline in health courses at a public university. A qualitative exploratory study was carried out using focus groups made up of tutors, preceptors and students. Once the categories and descriptors were defined, the 3P model was used for the content and results analysis. It is worth highlighting the redefinition of the teaching role, the students' perception of the challenges of interprofessional teaching, and the approximation with the health service. We correlate in the general discussion of this text lines of discussion with the person care, sociology, and pedagogy. The innovation proposed by the interprofessional discipline of Health Care followed the ideology of an international agenda. The reports from tutors, preceptors and students were a decisive stimulus to continue advancing in the other series of courses, in collaborative activities and interprofessional practices.

La reorientación de la formación profesional a través de políticas públicas ofreció la oportunidad para la creación de componentes curriculares para capacitar a los estudiantes en el ejercicio de prácticas interprofesionales. Este trabajo hace un análisis del contexto y de las implicaciones en la implementación de una disciplina interprofesional en cursos de salud en una universidad pública. Se realizó un estudio cualitativo exploratorio mediante grupos focales formados por tutores, preceptores y estudiantes. Una vez definidas las categorías y los descriptores, se utilizó el modelo 3P para el análisis de contenido y de resultados. Cabe destacar la redefinición del rol docente, la percepción de los estudiantes sobre los desafíos de la enseñanza interprofesional y la aproximación con el servicio de salud. Correlacionamos en la discusión general de este texto líneas de discusión con el cuidado de la persona, la sociología y la pedagogía. A inovação proposta pela disciplina interprofesional da Saúde seguiu a ideologia de uma agenda internacional. Los informes de tutores, preceptores y estudiantes fueron un estímulo decisivo para seguir avanzando en la otra serie de cursos, en actividades colaborativas y prácticas interprofesionales.

Time of clerkship rotations' interruption during COVID-19 and differences on Progress Test's scores.

OBJECTIVES: The transition from face-to-face to remote teaching is yet to be fully understood. In clinical training, traditional teaching must prevail because it is essential for the acquisition of skills and professionalism. However, the responses of each school to the pandemic and the decision on when to resume clerkship rotations were mixed. In this study, we aimed to analyze whether the time to resume clerkship rotations was associated with the performance of the students by using a multi-institutional Progress Test. METHODS: This is a cross-sectional study conducted at nine different Brazilian medical schools that administer the same annual Progress Test for all students. We included information from 1,470 clerkship medical students and analyzed the time of clinical training interruption as the independent variable and the student's scores as the dependent variable. RESULTS: The comparisons of the students' scores between the schools showed that there are differences; however, they cannot be attributed to the time the clerkship rotations were paused. The correlation between the schools' average scores and the time to resume clerkship rotations was not significant for the fifth year (r= -0.298, p=0.436) and for the sixth year (r= -0.440, p=0.240). By using a cubic regression model, the time to resume clerkship rotations could explain 3.4% of the 5-year students' scores (p<0.001) and 0.9% of the 6-year students, without statistical difference (p=0.085). CONCLUSIONS: The differences between the students' scores cannot be attributed to the time when the schools paused the clerkship rotations.

Time of clerkship rotations' interruption during COVID-19 and differences on Progress Test's scores

SUMMARY OBJECTIVES: The transition from face-to-face to remote teaching is yet to be fully understood. In clinical training, traditional teaching must prevail because it is essential for the acquisition of skills and professionalism. However, the responses of each school to the pandemic and the decision on when to resume clerkship rotations were mixed. In this study, we aimed to analyze whether the time to resume clerkship rotations was associated with the performance of the students by using a multi-institutional Progress Test. METHODS: This is a cross-sectional study conducted at nine different Brazilian medical schools that administer the same annual Progress Test for all students. We included information from 1,470 clerkship medical students and analyzed the time of clinical training interruption as the independent variable and the student's scores as the dependent variable. RESULTS: The comparisons of the students' scores between the schools showed that there are differences; however, they cannot be attributed to the time the clerkship rotations were paused. The correlation between the schools' average scores and the time to resume clerkship rotations was not significant for the fifth year (r= -0.298, p=0.436) and for the sixth year (r= -0.440, p=0.240). By using a cubic regression model, the time to resume clerkship rotations could explain 3.4% of the 5-year students' scores (p<0.001) and 0.9% of the 6-year students, without statistical difference (p=0.085). CONCLUSIONS: The differences between the students' scores cannot be attributed to the time when the schools paused the clerkship rotations.

Onze anos de Teste de Progresso na Unicamp: um estudo sobre a validade do teste / Eleven years of Progress Testing at Unicamp: a test validity study

Resumo: Introdução: O curso de Medicina da Universidade Estadual de Campinas (Unicamp) vem pondo à prova o aquisição cumulativa cognitiva de seus alunos por meio do Teste de Progresso (TP) há mais de uma década, de modo a possibilitar a análise da utilidade do exame como estratégia de apoio a decisões pedagógicas e apontar principais ameaças à validade dele. Objetivo: Este estudo teve como objetivos oferecer a análise de validade do TP e explicitar as oportunidades de utilização do teste especialmente para a determinação de padrões de suficiência cognitiva para a progressão no curso e ao final deste, e a identificação de estudantes em risco. Método: Trata-se de estudo observacional retrospectivo de uma série histórica de sucessivos testes escritos realizados para analisar o acúmulo cognitivo no período de 2006 a 2016, totalizando 11 anos e seis turmas consecutivas. Em cada momento de medida (aplicação do teste), o estudo utilizou um modelo misto, em que a exposição (realização do teste) e o desfecho (escore do teste) foram avaliados no mesmo ponto de tempo, o que caracteriza um estudo transversal (cross-sectional) cujos resultados sucessivos originarão as curvas de crescimento cognitivo Resultado: Observou-se um acúmulo cognitivo em torno de 6 pontos percentuais a cada nova testagem. Os estudantes ao completarem o sexto ano obtiveram um acerto de cerca de 65,7% (± 9,1). A cada testagem, determinou-se um "efeito piso" para identificar alunos com rendimento abaixo da média, que em geral se situou em cerca de 1,5 DP abaixo da média da respectiva turma. Conclusão: O TP-Unicamp oferece dados confiáveis para apoiar importantes decisões pedagógicas, tais como identificação de alunos em risco acadêmico por baixa performance, critérios para progressão e desempenho cognitivo ao final do curso. Como confiabilidade sofre influência da amostragem, e o aumento do número de itens de cada teste e o aumento da frequência de testagem podem ser estratégias a serem tomadas para superar essas limitações.

Abstract: Introduction: The Medical School of the State University of Campinas (UNICAMP) has been testing the cumulative cognitive acquisition among its students through the Progress Test (PT) for over a decade, making it possible to analyze the utility of the test as a strategy to support pedagogical decisions and to point out the main threats to its validity. Objective: To provide an analysis of the validity of the PT, explaining opportunities for its use, especially in determining standards of cognitive sufficiency for progression, standards of cognitive sufficiency at the end of the course, and identification of students at risk. Method: Retrospective observational study of historical series of successive written tests performed to analyze cognitive accumulation, covering a period from 2006 to 2016, totaling 11 years and 6 consecutive classes. At each instance of measurement (test application), the study uses a mixed model where exposure (test performance) and outcome (test score) are evaluated at the same time point, characterizing a cross-sectional study, the successive results of which will generate the cognitive growth curves. Result: A cognitive accumulation of around 6 percentage points was observed with each new test. Students, upon completing the 6th year, scored around 65.7% (± 9.1). A "floor effect" was determined for each test to identify students with below-average performance, which in general was about 1.5 SD below the average of the respective class. Conclusion: TP-Unicamp offers reliable data to support important pedagogical decisions, such as the identification of students at academic risk for low performance, criteria for progression and cognitive performance at the end of the course. As reliability is influenced by sampling, increasing the number of items in each test and increasing the frequency of tests could represent strategies to overcome potential limitations.

Exploring pooled analysis of pretested items to monitor the performance of medical students exposed to different curriculum designs.

Several methods have been proposed for analyzing differences between test scores, such as using mean scores, cumulative deviation, and mixed-effect models. Here, we explore the pooled analysis of retested Progress Test items to monitor the performance of first-year medical students who were exposed to a new curriculum design. This was a cross-sectional study of students in their first year of a medical program who participated in the annual interinstitutional Progress Tests from 2013 to 2019. We analyzed the performance of first-year students in the 2019 test and compared it with that of first-year students taking the test from 2013 to 2018 and encountering the same items. For each item, we calculated odds ratios with 95% confidence intervals; we also performed meta-analyses with fixed effects for each content area in the pooled analysis and presented the odds ratio (OR) with a 95% confidence interval (CI). In all, we used 63 items, which were divided into basic sciences, internal medicine, pediatrics, surgery, obstetrics and gynecology, and public health. Significant differences were found between groups in basic sciences (OR = 1.172 [CI95% 1.005 CI 1.366], p = 0.043) and public health (OR = 1.54 [CI95% CI 1.25-1.897], p < 0.001), which may reflect the characteristics of the new curriculum. Thus, pooled analysis of pretested items may provide indicators of different performance. This method may complement analysis of score differences on benchmark assessments.

Progress testing as a pattern of excellence for the assessment of medical students' knowledge - concepts, history, and perspective / Teste de progresso como padrão de excelência para avaliação de conhecimento dos alunos de medicina ­ conceitos, história e perspectivas

ABSTRACT: Progress test has been created with the necessity of an assessment method aligned with problem-based learning. Although it was specifically created to overcome the limitations of traditional assessment for problem-based learning, nowadays is used by different types of curricula. In this paper, we first present the basic assumptions, history, benefit and progress test challenges. Progress test overcomes many limitations of traditional assessment, such as validity and reliability. However, the implementation of a progress test is a logistical challenge. In addition, we discuss the limitation of progress tests when used as a summative assessment, which may not always be aligned with constructivist theory. When adding feedback and methods of analysis that consider multiple testing, progress test is then aligned with constructivist theory. Finally, the use of the progress test's sub scores may lack validity because of the low number of items; thus, pass/fail decision should not be based on the sub scores, but only on general scores. (AU)

RESUMO: O teste de progresso foi criado como um método de avalição alinhado a aprendizagem baseada em problemas. Apesar do teste de progresso ser criado especificamente para superar limitações da avaliação tradicional em currículos de aprendizagem baseada em problemas, hoje em dia, ele é utilizado em diferentes tipos de currículos. Nesse artigo, primeiro, apresentamos as premissas básicas, história, benefício e desafios do teste de progresso. O teste de progresso superou muitas limitações da avaliação tradicional, como os problemas de validade e confiabilidade. No entanto, a implementação do teste de progresso apresenta grandes desafios logísticos. Ademais, discutimos as limitações do teste de progresso quando utilizada de forma somativa, sendo que nem sempre é alinhada a teoria construtivista. Quando adicionado o feedback e métodos de análises que consideram múltiplos testes, o teste de progresso então se alinha a teoria construtivista. Finalmente, o uso das subcategorias do teste de progresso pode apresentar problemas de validade por causa do baixo número de item e consequentemente, decisões de aprovar ou reprovar não poderiam ser baseadas nas subcategorias, mas apenas na categoria geral (AU)

Volunteering among medical students during the COVID-19 pandemic: what should be taken into account? / Voluntariado entre estudantes de medicina durante a pandemia de COVID-19: o que devemos considerar?

Abstract: Introduction: during the COVID-19 pandemic, the universities suspended in-person activities and medical education adapted from the traditional format to virtual scenarios. Thus, the volunteering activities might improve the apprenticeship in critical situations and constitute a way to obtain competences and clinical skills. Objective: to identify the frequency of student volunteering during the COVID-19 pandemic and the associated factors. Methods: a cross-sectional study was performed targeting medical students in their last 3 years at a medical school in Brazil. Three hundred and fifty invitations were sent by email to ask students to participate in a self-administered, anonymous electronic survey during the pandemic. The primary outcome was the frequency of volunteer activities. Sociodemographic variables and characteristics of the medical students' activities were the dependent variables for the multivariate analysis that calculated the factors associated with volunteering. Results: One hundred and twenty-five respondents (35.8% response rate) were included in the analysis (no missing data). The frequency of volunteering was 52% and most of the participants were females (63.2%) and all had access to online activities. Telemedicine was the most frequent activity (56/65). After the multivariate analysis, it was found that a family income of 5.1-10 minimal wages (OR=2.32[0.94-6.42]), expressing the ability and confidence in a pandemic situation (OR=4.91[1.49-16.2]) and considering e-learning important before the pandemic (OR=16.46[1.35-200.32]) and exposure of more than 120 minutes to social media platforms were less motivating for volunteering. Conclusion: About half of the medical students volunteered during the COVID-19 pandemic. The presence of self-confidence, with previous training in a pandemic situation motivated the students to volunteer.

Resumo: Introdução: Durante a pandemia da Covid-19, as universidades suspenderam as atividades presenciais e o ensino médico adaptou-se do formato tradicional para os cenários virtuais. Assim, as atividades de voluntariado podem melhorar a aprendizagem em situações críticas e são uma forma de obter competências e habilidades clínicas. Objetivo: Este estudo teve como objetivo identificar a frequência do voluntariado nos estudantes durante a pandemia da Covid-19 e seus fatores associados. Método: Realizou-se um estudo transversal com estudantes de Medicina dos três anos finais de uma faculdade no Brasil. Trezentos e cinquenta alunos foram convidados via e-mail para uma pesquisa eletrônica anônima e autoaplicável, em meio à pandemia. O resultado primário foi a frequência das atividades voluntárias. Variáveis sociodemográficas e características da atividade dos estudantes foram as variáveis dependentes para análise multivariada dos fatores associados ao voluntariado. Resultado: Incluíram-se 125 alunos (taxa de resposta de 35,8%) na análise (sem dados ausentes). A frequência de voluntariado foi de 52%, a maioria dos participantes era do sexo feminino (63,2%), e todos tinham acesso a atividades on-line. A telemedicina foi a atividade mais frequente (56/65). Em análise multivariada, constatou-se que os seguintes fatores foram menos motivadores para o voluntariado: renda familiar entre 5,1 e dez salários mínimos (OR = 2,32 [0,94-6,42]), capacidade e confiança em atuar numa situação de pandemia (OR = 4,91 [1,49-16,2]), os alunos que consideraram importante uso do e-learning antes da pandemia (OR = 16,46 [1,35-200,32]) e exposição por mais de 120 minutos a plataformas de mídia social. Conclusão: Cerca de metade dos estudantes de Medicina foram voluntários durante a pandemia da Covid-10. A presença de autoconfiança com aconselhamento prévio em uma situação de pandemia motivou o aluno a se voluntariar.

Relationships between Bloom's taxonomy, judges' estimation of item difficulty and psychometric properties of items from a progress test: a prospective observational study.

BACKGROUND: Progress tests are longitudinal assessments of students' knowledge based on successive tests. Calibration of the test difficulty is challenging, especially because of the tendency of item-writers to overestimate students' performance. The relationships between the levels of Bloom's taxonomy, the ability of test judges to predict the difficulty of test items and the real psychometric properties of test items have been insufficiently studied. OBJECTIVE: To investigate the psychometric properties of items according to their classification in Bloom's taxonomy and judges' estimates, through an adaptation of the Angoff method. DESIGN AND SETTING: Prospective observational study using secondary data from students' performance in a progress test applied to ten medical schools, mainly in the state of São Paulo, Brazil. METHODS: We compared the expected and real difficulty of items used in a progress test. The items were classified according to Bloom's taxonomy. Psychometric properties were assessed based on their taxonomy and fields of knowledge. RESULTS: There was a 54% match between the panel of experts' expectations and the real difficulty of items. Items that were expected to be easy had mean difficulty that was significantly lower than that of items that were expected to be medium (P < 0.05) or difficult (P < 0.01). Items with high-level taxonomy had higher discrimination indices than low-level items (P = 0.026). We did not find any significant differences between the fields in terms of difficulty and discrimination. CONCLUSIONS: Our study demonstrated that items with high-level taxonomy performed better in discrimination indices and that a panel of experts may develop coherent reasoning regarding the difficulty of items.

Relationships between Bloom's taxonomy, judges' estimation of item difficulty and psychometric properties of items from a progress test: a prospective observational study

ABSTRACT BACKGROUND: Progress tests are longitudinal assessments of students' knowledge based on successive tests. Calibration of the test difficulty is challenging, especially because of the tendency of item-writers to overestimate students' performance. The relationships between the levels of Bloom's taxonomy, the ability of test judges to predict the difficulty of test items and the real psychometric properties of test items have been insufficiently studied. OBJECTIVE: To investigate the psychometric properties of items according to their classification in Bloom's taxonomy and judges' estimates, through an adaptation of the Angoff method. DESIGN AND SETTING: Prospective observational study using secondary data from students' performance in a progress test applied to ten medical schools, mainly in the state of São Paulo, Brazil. METHODS: We compared the expected and real difficulty of items used in a progress test. The items were classified according to Bloom's taxonomy. Psychometric properties were assessed based on their taxonomy and fields of knowledge. RESULTS: There was a 54% match between the panel of experts' expectations and the real difficulty of items. Items that were expected to be easy had mean difficulty that was significantly lower than that of items that were expected to be medium (P < 0.05) or difficult (P < 0.01). Items with high-level taxonomy had higher discrimination indices than low-level items (P = 0.026). We did not find any significant differences between the fields in terms of difficulty and discrimination. CONCLUSIONS: Our study demonstrated that items with high-level taxonomy performed better in discrimination indices and that a panel of experts may develop coherent reasoning regarding the difficulty of items.

Improvement of Faculty's Skills on the Creation of Items for Progress Testing Through Feedback to Item Writers: a Successful Experience / Melhora na Habilidade de Elaboração de Itens para o Teste de Progresso por meio de Feedback aos Autores: uma Experiência Exitosa

Abstract: The evaluation of knowledge is an important step in the teachers' work, because it is fundamental that the evaluation results be valid and reliable. Several faculty development programs have been undertaken to improve teachers' skills on item writing. Few studies, however, have explored the benefits of providing feedback to multiple-choice item writers. Progress Testing is a longitudinal assessment of students' knowledge gain. Their results are useful for generating feedback to students, teachers and medical school managers. The aim of this report is to present a successful experience of faculty development on high-quality item writing for Progress Testing through feedback about the fate of the items written by faculty members. Annually, faculties from medical schools participating in a consortium for the application of the Progress Test are asked to formulate new items to compose the test version. However, the fate of the items is not known by their authors, that is, they do not know if the items are incorporated into the test, if the items have good quality, what is the students' performance in each of the items, and what is the psychometric function of the items. In 2017, one of the schools participating in the consortium offered its item authors a feedback regarding the flaws in item writing, modification of items by the review committee and students' performance, as well as psychometric function in each item. The percentage of flawed item (items without enough quality to be selected for the test) was more than 30%. There was an inverse relation between the number of flawed item and the number of items included in the final exam, when analyzing the different content areas of the test (that is, Internal Medicine, Surgery, Pediatrics, Obstetrics and Gynecology, Public Health, and Basic Sciences). In the following year, we observed a decrease in the number of flawed item (less than 10%) and an increase in the number of eligible items to be selected for the Progress Testing. Therefore, giving feedback to item writers seems to be a good strategy for developing the faculty's ability on writing good quality items.

Resumo: A avaliação de conhecimentos é uma etapa importante do trabalho docente, pois é preciso que os resultados das avaliações sejam válidos e confiáveis. Diversos programas de desenvolvimento docente têm sido realizados para melhorar a habilidade em elaborar questões objetivas pelos professores. Poucos estudos, no entanto, exploraram os benefícios de fornecer feedback aos autores de itens de múltipla escolha. O Teste de Progresso é uma avaliação longitudinal do ganho de conhecimentos dos estudantes cujos resultados são úteis para gerar feedback aos próprios estudantes, aos professores e aos gestores das escolas médicas. O objetivo deste relato é apresentar uma experiência exitosa em gerar desenvolvimento docente na habilidade de elaborar itens de qualidade para o Teste de Progresso por meio do feedback a respeito da qualidade e do destino destes. Anualmente, os professores das escolas médicas participantes de um consórcio para aplicação do Teste de Progresso são solicitados a formular novos itens para compor a versão do teste. Porém, o destino dos itens não é conhecido por seus autores, ou seja, eles não sabem se as questões são incorporadas à prova, se elas têm boa qualidade, qual o desempenho dos estudantes em cada questão e qual o funcionamento psicométrico delas. Em 2017, uma das escolas participantes do consórcio ofereceu aos seus autores de questões uma devolutiva sobre as falhas na redação, a modificação dos itens pelo comitê revisor, o desempenho dos estudantes nas questões e o funcionamento psicométrico em cada item. A porcentagem de itens falhos, ou seja, de qualidade não satisfatória para a inclusão no teste, era superior a 30%. Houve uma relação inversa entre a quantidade de itens falhos e o número de itens incluídos na prova final, numa análise por área de conhecimento da prova (ou seja, clínica, cirurgia, pediatria, ginecologia e obstetrícia, saúde pública e ciências básicas). No ano seguinte, observamos uma diminuição no número de falhas de itens (menor que 10%) e um aumento no número de questões elegíveis a serem selecionadas para o exame de Teste de Progresso. Portanto, oferecer feedback para os redatores de questões parece ser uma boa estratégia para desenvolver a habilidade docente em elaborar itens objetivos de boa qualidade.

The Validity Concept in Medical Education: a Bibliometric Analysis / O Conceito de Validade na Educação Médica: uma Análise Bibliométrica

Abstract: Introduction: Assessment is a critical part of learning and validity is arguably its most important aspect. However, different views and beliefs led to a fragmented conception of the validity meaning, with an excessive focus on psychometric methods and scores, neglecting the consequences and utility of the test. The last decades witnessed the creation of a significant number of tests to assess different aspects of the medical profession formation, but researchers frequently limit their conclusions to the consistency of their measurements, without any further analysis on the educational and social impacts of the test. The objective of this work is to determine the predominant concept of validity in medical education assessment studies. Method: The authors conducted a bibliometric research of the literature about studies on the assessment of learning of medical students, to determine the prevalent concept of validity. The research covered a period from January 2001 to august 2019. The studies were classified in two categories based on their approach to validity: (1)" fragmented validity concept" and (2)" unified validity concept". To help with validity arguments, the studies were also classified based on Miller's framework for clinical assessment. Results: From an initial search resulting in 2823 studies, 716 studies were selected based on the eligibility criteria, and from the selected list, of which 693 (96,7%) were considered studies of the fragmented validity concept, which prioritized score results over an analysis of the test's utility, and only 23 studies (3,2%) were aligned with a unified view of validity, showing an explicit analysis of the consequences and utility of the test. Although the last decade witnessed a significant increase in the number of assessment studies, this increase was not followed by a significant change in the validity concept. Conclusions: This bibliometric analysis demonstrated that assessment studies in medical education still have a fragmented concept of validity, restricted to psychometric methods and scores. The vast majority of studies are not committed to the analysis about the utility and educational impact of an assessment policy. This restrictive view can lead to the waste of valuable time and resources related to assessment methods without significant educational consequences.

Resumo: Introdução: Avaliação é uma parte crítica da aprendizagem, e validade é sem dúvida seu aspecto mais importante. No entanto, diferentes visões e crenças levaram a uma concepção fragmentada do significado de validade, com um foco excessivo nos métodos psicométricos e escores, negligenciando a utilidade do teste. As últimas décadas testemunharam a criação de um número significativo de testes para avaliar diferentes aspectos da formação da profissão médica, mas os pesquisadores frequentemente limitam suas conclusões à consistência de suas medidas, sem nenhuma análise adicional sobre os impactos educacionais e sociais do teste. O objetivo deste trabalho é determinar o conceito predominante de validade nos estudos de avaliação em educação médica. Método: Foi realizada uma pesquisa bibliométrica da literatura de estudos sobre avaliação da aprendizagem de estudantes de Medicina para determinar o conceito prevalente de validade. A pesquisa abrangeu o período de janeiro de 2001 a agosto de 2019. Os estudos foram classificados em duas categorias: 1. "conceito de validade fragmentada" e 2. "conceito de validade unificada". Para ajudar nos argumentos de validade, os estudos também foram classificados com base na estrutura de Miller para avaliação clínica. Resultados: A partir de uma pesquisa inicial que resultou em 2.823 estudos, selecionaram-se 716 com base nos critérios de elegibilidade, e consideraram-se 693 (96,7%) estudos com conceito fragmentado de validade que priorizavam os resultados dos escores em detrimento de uma análise da utilidade do teste, e apenas 23 (3,2%) foram alinhados com uma visão unificada de validade, apresentando uma análise explícita das consequências e da utilidade do teste. Embora a última década tenha testemunhado um aumento expressivo de estudos sobre avaliação, esse crescimento não foi acompanhado por uma mudança significativa do conceito de validade. Conclusões: Esta análise bibliométrica demonstrou que os estudos sobre avaliação de aprendizagem em educação médica têm um conceito fragmentado de validade, limitados aos métodos psicométricos e escores. A grande maioria dos trabalhos não está comprometida com uma análise sobre a utilidade e o impacto educacional de uma política de avaliação. Essa visão restritiva pode levar à perda de tempo e recursos valiosos com métodos de avaliação sem consequências educacionais significativas.

Teste de Progresso em Consórcios para Todas as Escolas Médicas do Brasil / Consortia of Cross-Institutional Progress Testing for All Medical Schools in Brazil

RESUMO O Teste de Progresso é uma ferramenta de avaliação longitudinal do ganho de conhecimento de estudantes que tem sido aplicada no Brasil há mais de dez anos. O teste situa o estudante em seu processo evolutivo de ensino-aprendizagem e permite à instituição realizar o diagnóstico de suas deficiências ao longo da estrutura curricular. Ele pode ser utilizado pelos colegiados competentes para avaliação de alterações curriculares e avaliações específicas de disciplinas ou módulos de ensino. Com base na experiência de um consórcio de escolas, a Associação Brasileira de Educação Médica (Abem) propôs um projeto que tinha como um de seus objetivos incentivar escolas de todo o País a adotarem o Teste de Progresso como uma de suas ferramentas de avaliação. Reportamos a estratégia adotada para constituir núcleos interinstitucionais de avaliação com Teste de Progresso, bem como os resultados da primeira prova nacional do Teste de Progresso, que contou com a participação de 58 escolas e 23.065 estudantes. A implantação de núcleos interinstitucionais de Teste do Progresso com processos colaborativos de realização da prova representou um avanço para as escolas envolvidas. As escolas iniciaram um processo de colaboração não apenas para o Teste de Progresso, mas também para o intercâmbio de informações e experiências que trocam com base no conhecimento de cada uma. O projeto funcionou como o início de um movimento para que escolas médicas de todas as regiões do País adotem o Teste de Progresso como uma ferramenta de avaliação com potencial para reorientar a formação médica, ao fornecer um diagnóstico de formação em nível individual e institucional.

ABSTRACT Progress Testing has been applied in Brazil for more than ten years. It is a tool for longitudinal evaluation of students' knowledge across levels of training. The test situates the student in his evolutionary teaching-learning process and allows the institution to diagnose its deficiencies throughout the curricular structure. It can be used to evaluate curricular changes and specific assessments of disciplines or teaching modules. From the experience of a consortium of schools, the Brazilian Association of Medical Education (ABEM) proposed a project that aimed to encourage schools throughout the country to adopt Progress Testing as one of its evaluation tools. We report on the strategy adopted to establish interinstitutional evaluation nuclei with Progress Testing, as well as the results of the first national exam, administered by 58 schools to 23,065 students. The implantation of cross-institutional Progress Testing triggered collaborative processes among the schools involved. Schools have begun a collaborative process not only for the Progress Testing, but also for the exchange of information. The project served as the beginning of a movement for medical schools in all regions of the country to adopt Progress Testing as an assessment tool with the potential to reorient medical training by providing information regarding institutional and individual performances.

The Correlation Between Students' Progress Testing Scores and Their Performance in a Residency Selection Process.

Brazil is currently seeing an increased number of medical schools, leading to high competition for medical residency vacancies. Public managers have thus considered Progress Testing scores potentially useful as part of the final decision in the medical residency selection process. We analyzed whether there is a correlation between students' Progress Testing scores and their performances in medical residency selection. We examined four subsequent cohorts of students who attempted Progress Testing yearly and compared their accumulated scores with their medical residency selection scores from Botucatu Medical School, Universidade Estadual Paulista. We included 212 students who finished the 6-year medical course in 2013, 2014, 2015, and 2016. The comparison between the area under the Progress Testing curve and the medical residency selection score was performed using a Pearson correlation, with a p value set at < 0.05. We found a positive association between the two scores (p < 0.05 for the 4 years). Next, the students were grouped according to their performance in Progress Testing: above one, within one, and below one standard deviation. A chi-square test was used to compare the rates of approval with the second step of the medical residency selection process. Approval rates were 91.7%, 69.2%, and 42.1%, respectively (p < 0.05). We conclude that, in fact, there is a correlation between students' performance on these measures. This is partially explained by the fact that both instruments measure cognitive competencies and knowledge. These data may support national policy changes for medical residency selection.

A heterogeneidade na correção dos portfólios no curso de Medicina: percepção discente / Heterogeneity in the correction of portfolios in the Medicine course: student perception / La heterogeneidad en la corrección de los portafolios en el curso de Medicina: percepción discente

Na Faculdade de Ciências Médicas da Unicamp, o portfólio foi implementado no Curso de Medicina em 2001, - é desenvolvido no quarto ano, quando os discentes atuam nas Unidades Básicas de Saúde. Cada portfólio é corrigido por um docente, e, após as considerações necessárias de cada tópico, atribui-se uma nota. O objetivo deste estudo foi compreender a percepção dos discentes sobre a correção do portfólio pelo grupo docente. Trata-se de um estudo descritivo, com análise qualitativa dos dados. Foram realizadas 31 entrevistas semiestruturadas, após a devolutiva dos portfólios. Os dados foram consolidados utilizando-se a análise de conteúdo proposta por Bardin. Os alunos consideraram que a avaliação por meio do portfólio necessita ser aprimorada com definição clara dos critérios de correção. Na percepção discente, destaca-se a heterogeneidade e a subjetividade dos docentes nessa correção.

The portfolios were implemented in the Medicine Course of the School of Medical Sciences of Unicamp in 2001; it is developed in the fourth year, when the students work in basic health units. Each portfolio iss corrected by a professor, and after the necessary considerations on each topic, a grade is awarded. This study aimed at understanding the perception of students regarding the correction of the portfolios by the teaching staff. It consisted of a descriptive study with qualitative data analysis. Thirty-one semi-structured interviews were conducted after the portfolios were returned. Data were consolidated with the use of the content analysis proposed by Bardin. The students considered that the assessment through the portfolio needs to be improved with a clear definition of the correction criteria. The heterogeneity and subjectivity of the professor in such correction is highlighted in the student perception.

En la Facultad de Ciencias Médicas de Unicamp, el portafolio se implementó en el curso de Medicina en 2001 y se desarrolla en el cuarto año, cuando los discentes actúan en las Unidades Básicas de Salud. Cada portafolio se corrige por un docente y después de las consideraciones necesarias de cada tópico se atribuye una nota. El objetivo de este estudio fue comprender la percepción de los discentes sobre la corrección del portafolio por el grupo docente. Se trata de un estudio descriptivo con análisis cualitativo de los datos. Se realizaron 31 entrevistas semi-estructuradas, después de la devolución de los portafolios. Los datos se consolidaron utilizándose el análisis de contenido propuesto por Bardin. Los alumnos consideraron que la evaluación por medio de portafolio tiene que perfeccionarse con definición clara de los criterios de corrección. En la percepción discente, se destaca la heterogeneidad y la subjetividad de los docentes en esa corrección.

Trajetória profissional de egressos do curso de Medicina da Universidade de Campinas (Unicamp), São Paulo, Brasil: o olhar do ex-aluno na avaliação do programa / Trayectoria profesional de egresados del curso de Medicina da Universidad de Campinas, São Paulo, Brasil: la mirada del ex-alumno en la evaluación del programa / Professional trajectory of medical course alumni from from Campinas University, São Paulo, Brazil: graduates' point of view in evaluating the course

A Faculdade de Ciências Médicas da Unicamp implantou uma ampla reforma curricular, que antecedeu as Diretrizes Curriculares Nacionais (DCN) de 2001. Analisou-se a trajetória profissional e o impacto da reforma curricular na formação pela percepção dos egressos. Alunos concluintes de 1991-2005 compuseram o grupo pré-reforma e os de 2006-2012, o grupo pós-reforma, que receberam um questionário semiestruturado on-line por e-mail e redes sociais. Obtiveram-se 213 (15,8%) respostas do grupo pré e 126 (16,1%) do pós-reforma. Os dados apontaram pouca dispersão geográfica dos ex-alunos, ocupação de 2,4 postos de trabalho por médico, entendimento da residência como necessária, maior frequência a cursinhos preparatórios no grupo pós-reforma e sugestão de inclusão de administração/gestão em saúde no currículo. A integração entre áreas básicas/clínicas apresentou significante melhora pós-reforma. A percepção sobre a graduação foi altamente positiva. O estudo contribuiu para implantar o acompanhamento sistemático de egressos no curso de Medicina.(AU)

La Facultad de Ciencias Médicas de Unicamp implantó una amplia reforma curricular que antecedió a las directrices curriculares nacionales de 2001. Se analizó la trayectoria profesional y el impacto de la reforma curricular en la formación a partir de la percepción de los egresados. Alumnos concluyentes de la promoción 1991-2005 compusieron el grupo pre-reforma y los de 2006-2012 el grupo post-reforma, recibiendo un cuestionario semi-estructurado, on line, por e-mail y redes sociales. Se obtuvieron 213 (15,8%) respuestas del grupo pre-reforma y 126 (16,1%) del post-reforma. Se encontró poca dispersión geográfica de los ex-alumnos, ocupación de 2,4 puestos de trabajo/médico, comprensión de la residencia como necesaria, mayor frecuencia a cursos preparatorios en el grupo post-reforma y sugerencia de inclusión de administración/gestión en salud en el currículo. La integración entre áreas básicas/clínicas presentó una mejora significativa post-reforma. La percepción sobre la graduación fue altamente positiva. El estudio contribuyó para implantar el acompañamiento sistemático de egresados en el curso de Medicina.(AU)

The School of Medical Sciences of UNICAMP implemented a new curriculum in 2001, predating the national guidelines. This study analyzed the professional trajectory and the new curriculum's impact. Graduates from 1991 to 2005 were grouped in the pre-curricular renovation group, and those from 2006 to 2012 in the post-group. A semi-structured survey was sent through e-mail and social media. There were 213 (15.8%) answers from pre-renovation group and 126 (16.1%) from post. Data showed low geographic dispersion of alumni, 2.4 work posts per physician, residency as a necessary complementation, suggestions for curricular inclusion of management issues and higher frequency of preparatory courses for residency in post-renovation group. There are signs of better integration between basic and clinical areas after the reform. Alumni showed positive perceptions regarding their undergraduate course. The study contributes to the implementation of systematic follow-up of graduates.(AU)